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With the widespread dissemination of picture archiving and communication systems (PACSs) in hospitals, the amount of im- 
aging data is rapidly increasing. Effective image retrieval systems are required to manage these complex and large image da- 
tabases. The authors reviewed the past development and the present state of medical image retrieval systems including text- 
based and content-based systems. In order to provide a more effective image retrieval service, the intelligent content-based 
retrieval systems combined with semantic systems are required. 
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I. Introduction 

With the widespread use of picture archiving and communi- 
cation system (PACS) in the hospitals, the amount of medi- 
cal image data is rapidly increasing. Thus, the more efficient 
and effective retrieval methods are required for better man- 
agement of medical image information. 

There are two ways that medical images are retrieved, text- 
based and content-based methods [1]. In text-based image 
retrieval, the images are retrieved by the manually annotated 
text descriptions and traditional database techniques to 
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manage images are used. In content-based image retrieval, 
the images are retrieved on the basis of features such as 
color, texture, shape, and so on, which were derived from the 
images themselves. 

So far, a variety of medical image retrieval systems have 
been developed using either method (text -based or content- 
based) or combining two methods; a rough classification of 
medical image retrieval methods is displayed in Figure 1. 
Therefore, we described two technological approaches for 
medical image retrieval - past development and present sta- 
tus, and future perspectives. 

II. Text-Based Medical Image Retrieval 

Text-based image retrieval system can be traced back to 
1970s. Text-based image retrieval system is prevalent in the 
search on the internet web browsers. Although text-based 
methods are fast and reliable when images are well anno- 
tated, they cannot search in unannotated image databases. 
Moreover, text-based image retrieval has the following ad- 
ditional drawbacks, it requires time-consuming annotation 
procedures and the annotation is subjective [2]. Text-based 
query commonly results in irrelevant images (Figure 2). 
Thus, to support effective image searching, retrieval methods 
based on the image content were developed. 
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III. Content-Based Medical Image Retrieval 

In content-based image retrieval systems, images are indexed 
and retrieved from databases based on their visual content 
(image features) such as color, texture, shape, etc. Com- 
mercial content-based image retrieval systems have been 
developed, such as QBIC [3], Photobook [4], Virage [5], Vi- 
sualSEEK [6] , Netra [7] . Eakins [8] has divided these image 
features into three levels as followings: 1) Level 1 - Primitive 
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Figure 1. Classification of medical image retrieval methods. 



features such as color, texture, shape or the spatial location 
of image elements. Typical query example is 'find pictures 
like this'; 2) Level 2 - Derived attributes or logical features, 
involving some degree of inference about the identity of the 
objects depicted in the image. Typical query example is 'find 
a picture of a flower'; 3) Level 3 - Abstract attributes, involv- 
ing complex reasoning about the significance of the objects 
or scenes depicted. Typical query example is 'find pictures of 
a beautiful lady' 

The majority of content-based image retrieval systems 
mostly offer level 1 retrieval, a few experimental systems 
level 2, but none level 3. 

Commonly used image features for content-based image 
retrieval were followings: 1) Color: color is one of the visual 
cues often used for content description, but most medical 
images are grayscale. Thus, color features are not used for 
medical image retrieval; 2) Texture: texture features mean 
spatial organization of pixel values of an image and used in 
standard transform domain analysis by tools such as Fou- 
rier transform, wavelets, Gabor or Stockwell filters. In the 
medical images, texture features are useful because they can 
reflect the details within an image structure; 3) Shape: shape 
feature has broad range of visual cues such as contour, curve, 
surfaces, and so on. Recently, many methods measures simi- 
larity between images using shape features has been devel- 
oped. 




Figure 2. Text-based query commonly 
retrieves irrelevant im- 
ages on the internet web 
browser. For user query 
"bike (bicycle)," irrelevantly 
retrieved images (not rel- 
evant to bike) are shown 
because image annotations 
contain a word "bike" such 
as bike tour, school bike 
rack, water bike, abbrevia- 
tion, etc. (upper two layers). 
Miscellaneous medical im- 
ages retrieved by query of 
"Reconstruction and com- 
puted tomography (CT)" are 
displayed in the lower 3rd 
layer of the figure (CT for 
image reconstruction, 3-di- 
mensional reconstruction 
CT image, CT reconstruction 
algorithm, bone reconstruc- 
tion with CT, breast recon- 
struction using CT). 
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Formerly developed commercial content-based image re- 
trieval systems characterized images by global features such 
as color histogram, texture values and shape parameters, 
however, for medical images, the systems using global im- 
age features failed to capture the relevant information [9]. In 
medical images, the clinically useful information is mostly 
highly localized in small areas of the images, that is, the ratio 
of pathology bearing pixels to the rest of the images is small. 
Thus, the global image features such as color, texture, shape, 
etc. cannot effectively characterize the content of the medical 
images. 



Initially, medical images were included in the content- 
based retrieval systems as a subdomain for trials [10,11]. 
Recent trials for content-based medical image retrieval were 
ASSERT system [12] for high resolution computed tomog- 
raphy (CT) images of the lung and image retrieval for medi- 
cal applications (IRMA) system [13] for the classification 
of images into anatomical areas, modalities and viewpoints. 
Flexible image retrieval engine (FIRE) system handles dif- 
ferent kinds of medical data as well as non-medical data like 
photographic databases [14]. 

In ASSERT system, the system lets the user extract pafhol- 
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Figure 3. Image retrieval for medical 
applications (IRMA) con- 
tent-based image retrieval 
results (http://irma-project. 
org/). 
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Figure 4. Flexible image retrieval en- 
gine (FIRE) content-based 
image retrieval results (http:// 
thomas.deselaers.de/fire/). 
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ogy-bearing regions in lung images and these regions are 
then characterized by their grayscale, texture, and shape at- 
tributes, which are used for image retrieval. 

The IRMA system (Figure 3) is a generic web-based x-ray 
retrieval system. It allows the user to extract images from a 
database given an x-ray image query. Local features and sim- 
ilarity measures are used to compute the nearest images. This 
system also developed a classification code for medical im- 
ages to classify medical images based on four axes (modality, 
body orientations, body region, and biological system) [15]. 
The local features are derived from the previously classified 
and registered images that have been segmented automati- 
cally. IRMA analyzes content of medical images using a six- 
layer information model: raw data, registered data, feature, 
scheme, object, and knowledge. IRMA lacks the ability for 
finding particular pathology that may be localized in par- 
ticular regions within the image. 

In FIRE system (Figure 4), query by example image is im- 
plemented using a large variety of different image features 
that can be combined and weighted individually and rel- 
evance feedback can be used to refine the result [ 16] . 

The integration of content-based image retrieval system 
into the PACS system has been proposed [17,18]. However, 
an effective and precise medical image retrieval still remains 
a problem and recent researches aim at developing tech- 



niques that overcome this point. 

IV. Controlled Vocabulary-Based System 
for Medical Image Retrieval 

Currently, work is underway to create tools to enable seman- 
tic annotation of images using ontologies, providing an op- 
portunity to enhance content-based image retrieval systems 
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Figure 5. CliniClue Systematized Nomenclature of Medicine - 
Clinical Terms (SNOMED-CT) web browser (http://www. 
cliniclue.com/). 



Foundational Model Explorer Options | Help 



Select navigation tree ripe: [part 



♦ Left lung 

--Pjglli JUllg 

'Lung 



4- Lung parenchyma 
♦Visceral pleura 

= Inrrapuiiioi; i' hi >' " i'iilir.-."iint . . 
■ Intrapuknonary part of bronchial v< 

- Anterior pulmonary nerve plexus 

+ Pleural sac 
-J 1 Cardiovascular svsfeiu of male human body 
♦Alimentary system 
->Male urinarv system 
'Male feiutal system 
-> Endocrine system 



-'Female bodv proper 
♦Neck 
-■Female trunk 

-Thoracic segment of female trunk 
"Female thorax 

♦ Anterior female chest 
♦Posterior chest 
4> Female chest wall 
"Compartment of thorax 

• Right pulmoplcural compartment 
->Left pukuopleural compartment 

- Content of thorax 



■ Content of compartment of right hernithorax 
• Content of compartment of left hemitliorat 
. Er, do thoracic fascia 
"Set of thoracic viscera 

♦Heart 

♦ Pericardial sac 
♦Right lung 

4- Left kmg 

♦ Pleural sac 
+Thvmus 
-Li 



♦Long parenchyma 



■ Intrapulmonajy part of bronchial arter. 1 
♦ Pulmonary lymphatic chain 



PREFERRED NAME: ❖ 



Neural network of lobe of lung 
Vasculature of lobe of long 



PART OF: <► 

REGIONAL PART: <► 

Bronchopulnroaarv segruen 



REGIONAL PART OF: <► 
|L"e| 

CONSTITUTIONAL PART: * 



| Parenchyma of lobe of long 



etn orkoflobe of lung 
are of lobe ol lung 



HAS DIMENSION: 0 



HAS BOUNDARY: <► 



HAS INHERENT 3-D SHAPE: O 



Figure 6. Foundational model explorer 
browser (http://fme.biostr. 
washington.edu/FME/index. 
html/). 
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with richer descriptions of images. The suggested ontology 
systems are controlled terminologies such as Systematized 
Nomenclature of Medicine - Clinical Terms (SNOMED-CT) 
[19], Foundational Model of Anatomy (FMA) [20] and Ra- 
diology Lexicon (RadLex) [21]. 

1. SNOMED-CT Terminology 

SNOMED-CT aims to be a comprehensive terminology that 
provides clinical content and expressivity for clinical docu- 



Medical Image Retrieval 

mentation and reporting [19]. The SNOMED hierarchy is 
easy to compute, which was the primary reason for selecting 
the terminology for the research. SNOMED-CT has approxi- 
mately 370,000 concepts and 1.5 million triples i.e., relation- 
ships of one concept with another in the terminology (Figure 5). 

2. FMA 

FMA is a reference ontology for the terms of anatomy and 
developed and maintained by the University of Washington 
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and the US National Library of Medicine [20]. It contains 
71,202 anatomical entities and more than 1,500,000 relations 
which form the machine processable standard vocabulary. 
FMA also provides definitions for conceptual attributes, 
part-whole, location, and other spatial associations of ana- 
tomical entities (Figure 6). 

3. RadLex Terminology 

RadLex was developed to create a unifying source for 
medical imaging terminology by the Radiological Society 
of North America [22] and currently contains more than 
32,000 standardized terms used in radiology reports (Figure 7). 
It contains not only domain knowledge but also lexical in- 
formation such as synonymy. RadLex terminology helps the 
analysis of radiological information, allows uniform index- 
ing of image databases, and enables structuring medical im- 
age information [23,24] . 

V. Combined Text and Content-Based 
Medical Image Retrieval 

Considering the intrinsic difference between the text and 
image in representing and expressing information, there 
have been approaches to combine the text-based and con- 
tent-based image retrieval. Techniques that perform the 
text-based method first [25,26] and two methods at the same 
time [27,28] were studied. 

The hybrid image retrieval systems to incorporate external 
knowledge that is encoded in lexicons, thesauri and ontolo- 
gies were suggested [29] . 

The new project, Annotation and Image Markup (AIM) for 
medical image annotation and markup is being developed 
to make all the key semantic (Figure 8) content of images 
machine-readable using controlled terminologies (mainly 
RadLex) and image markup standards [30] . 

VI. Conclusion 

In the domain of medical imaging informatics, a huge 
amount of image data is being produced. A lot of work has 
already been done to improve the image retrieval systems. 
One is text-based approach and the other is content-based. 
Each method has its own advantages and disadvantages. 
Text-based method is widely used and fast, but it requires 
precise annotation. Content-based approach provides se- 
mantic retrieval, but effective and precise techniques still 
remains elusive. 

Recently, a new controlled vocabulary, RadLex was devel- 
oped to provide standardized terms for images and corn- 
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bined text and content-based methods were developed. For 
improved semantic image retrieval, it is proposed that image 
retrieval techniques be effectively integrated with external 
knowledge, annotation tools, and image markup systems. 

In the near future, it is expected that the semantic contents 
of medical images will be totally computationally- accessible 
and reusable by the application of ontology and the develop- 
ment of new convenient tools 
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